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Executive Summary 


Background 


During the summer of 2021, Montgomery County Public 
Schools (MCPS) offered a variety of voluntary summer 
school programs to elementary, middle, and high school 
students using both online and in-person models. The 
summer programs were designed to maximize learning 
recovery and mitigate learning loss during summer break in 
light of the COVID-19 pandemic’s disruption starting in 
March 2020. This report examines the impact of the 2021 
summer programs on students’ mathematics and literacy 
skills using pretests and posttests for the summer program, 
fall and spring 2021 Measures of Academic Progress in 
Reading/Reading Fluency (MAP-R/RF) and Mathematics 
(MAP-M). Online surveys were used to gather information 
on the impact of the summer program on social-emotional 
learning (SEL) for middle and high school students. 


Research Questions 


1. What is the demographic composition of students who 
participated in the 2021 summer programs? 

2. What impact did the summer school programs have on 
students’ academic outcomes? 

3. To what extent did the 2021 summer school programs 
mitigate students’ academic learning disruption? 

4. What impact did the summer school program have on 
students’ social-emotional learning (SEL)? 


Key Findings 


Participation in 2021 summer programs varied by grade 
and student groups 


e =Nearly 30 percent (N=44,552) of students enrolled in the 
2021-2022 school year attended the 2021 summer 
programs. These students were distributed across the 
elementary (51%), middle school (20%), and high school 
(29%) levels. 


e Summer program participation rates varied by school 
level and student groups, ranging from 45% among 
elementary primary Black or African American students 
receiving Free and Reduced Meals Services (FARMS) to 
20% among middle school Asian/White and All other 
students not receiving FARMS services. 


The student surveys showed that students reported many 
socioemotional benefits 


e Two-thirds reported they learned strategies to build 
positive relationships and collaboration. 


e = Thirty eight percent of the 8,020 students recommended 
for the summer program attended. The participation rate 
for recommended students varied from 12.2% for Grade 
8 to 53.6% for Grade 11 students. 


e Overall, students attended 92% of their summer 
program. Middle and high school students attended 98% 
and 99% of time scheduled for their programs. 


The summer school program had a positive impact on 
students’ academic outcomes 


e Students in Grades 2—8 who attended the 2021 summer 
programs had significantly higher mathematics scores on 
the posttests across grade levels and student groups, 
compared to their average pretest scores. 


e Grade 2-5 participants attained statistically higher 
literacy scores on the posttests than on the pretests across 
grade levels and student groups. 


e Middle school students experienced gains in posttest 
literacy scores relative to pretests, and these gains were 
Statistically significant for Grade 6 and 8 students 
(P<0.05). 


Overall, the fall MAP scores for participants and 
nonparticipants were similar but varied by grade levels, 
racial/ethnic groups, and special services 


e Although the result of this analysis showed no evidence 
that participants outperformed nonparticipants on the fall 
MAP assessments, it demonstrated that the summer 
school program contributed to maintaining the spring 
level of performance. Specifically, both participants and 
nonparticipants either remained steady or experienced 
minor and comparable changes (upward or downward) 
from spring to fall. 


e While taking into consideration the spring performance 
and demographical variables, the between-group 
difference in the fall performance was statistically 
significant for some grades and student groups. 
However, the between-group differences were 
negligible, showing that the performance of the summer 
program participants and  nonparticipants was 
comparable in the spring and fall. 


e The majority of the survey respondents reported that they 
learned or strengthened their problem-solving and time 
management skills. 

e Over four-fifths reported that their summer school 
program provided a positive learning environment. 
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Examining Impact of MCPS 2021 Summer Program 


Background 


During the summer of 2021, Montgomery County Public 
Schools (MCPS) offered a variety of voluntary summer 
school programs to elementary, middle, and high students 
using both a virtual and in-person model. The goal of the 
summer programs was to maximize learning recovery and 
mitigate learning loss in light of the COVID-19 pandemic’s 
disruption starting in March 2020 (MCPS, 2021a). This 
report examines the impact of the 2021 summer programs on 
students’ mathematics and literacy skills using data from 
pretest, posttest, and social-emotional learning (SEL) 
measures. In addition, fall 2021 Measures of Academic 
Progress in Reading and Mathematics (MAP-R and MAP-M) 
were also used to determine whether the summer school 
program helped facilitate students learning over the summer. 


Purpose 


The purpose of this study was to examine the extent to which 
the 2021 summer school programs supported the learning 
recovery efforts in literacy and mathematics for MCPS 
students. Specifically, this study examined the impact of the 
2021 summer school programs on students’ academic 
outcomes and SEL. 


This study examined the differences in participants’ 
performance in mathematics and literacy by the end of the 
summer; report of social-emotional learning, and the summer 
programs’ influence on participants’ performance on the fall 
MAP, comparing the performance of students in Grades 3, 6, 
and 9 with a matched sample of peers who did not participate 
in the summer school program. 


Description of the 2021 Summer School Programs 


The summer school programs aimed to enable all students to 
engage in summer academic and special/elective courses 
(MCPS, 2021a). MCPS offered these summer programs at no 
cost and made transportation services available to students 
from all schools and clusters. The MCPS 2021 voluntary 
summer programs were delivered either in person or 
virtually. The programs focused on helping students recover 
and facilitate their learning following the 2020—2021 school 
year in mathematics, literacy, SEL, and foundational skills to 
help prepare them for future success (e.g., teamwork, 
collaboration, critical thinking skills). 


The programs delivered virtually were offered centrally, with 
student enrollment across many schools. Local programs 
offered in-person instruction at the home school. 
Furthermore, each school had the flexibility to adapt the 
academic and special/elective courses to meet the needs of 
students. The summer programs ran for 19 days (July 6—30, 
2021) for approximately 4.5 hours per day, except for 31 Title 
I schools (MCPS, 2021a; MCPS, 2021b), which ran for 24 
days (July 6—August 6, 2021; see Appendix A1). At the start 
of the summer program, students took a diagnostic 
assessment or pretest. At the end of the program, students 
took a posttest to reassess their mathematics and literacy 
skills. 


Funding for summer school programs was provided through 
the federal American Rescue Plan Elementary and Secondary 
School Emergency Relief (ARP ESSER) Fund (Office of 
Elementary and Secondary Education, 2021). 


Significance of the study 


This evaluation supports MCPS’s strategic priority of 
mitigating learning disruption and accelerating learning 
recovery across schools stipulated in the MCPS Strategic 
Plan (MCPS, 2021b) and the Maryland State Department of 
Education’s (MSDE) requirements that each local school 
system a) establish and implement a summer school program 
for public school students to address the effects of the 
COVID-19 pandemic on education; and b) evaluate the 
program’s effectiveness at the conclusion (MSDE, 2021, p. 
3). These findings are expected to inform instructional 
decisions for improving academic achievement. 
Furthermore, the results from this study will help with future 
summer program planning. 


Literature Review 


Disruption of Learning and Assessment 


The COVID-19 pandemic created the most extensive 
disruption of education systems in human history, affecting 
more than 1.6 billion learners in more than 200 countries 
(Pokhrel & Chhetri, 2021). Beginning in March 2020, school 
systems across the country faced extraordinary challenges in 
educating students at a distance. The abrupt transition from 
traditional face-to-face learning to virtual settings was an 
entirely new experience for both students and educators. 
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However, unlike the economic and healthcare sectors, which 
regularly updated the public on the pandemic’s impact, 
school systems do not typically share student performance 
data with the public at highly-frequent intervals. As a result, 
limited access to and updates on student performance and 
state summative assessment data resulted in considerable 
apprehension among parents, educators, and policymakers. 
Furthermore, during the 2020—2021 school year, there was no 
requirement to collect any data that might be a proxy for 
achievement after all 50 states were permitted to skip the 
statewide standardized tests required by federal law 
(Gewertz, 2020; US Department of Education, 2020). 


Impact of COVID-19 on Learning 


At the start of the 2020-2021 school year, researchers 
projected that returning students would start fall 2020 with 1) 
approximately 63%-—68% of the learning gains in reading, 2) 
37%-50% of the learning gains in mathematics relative to a 
typical school year, and 3) only a third of students would 
potentially make significant gains in reading (Kuhfeld, 
Soland, Tarasawa, Johnson, Ruzek, & Lewis, 2020). 
However, these researchers also cautioned that the 
projections were inherently speculative, and could very well 
have over or underestimated the impact COVID-19 had on 
students learning. 


More recent reports concur that many students made little or 
no progress while learning from home (Mcmurdock, 2021; 
MSDE, 2021; Office of Civil Rights, 2021; Pokhrel & 
Chhetri, 2021; Dorn, Hancock, Sarakatsannis, & Viruleg, 
2020; Curriculum Research Associates, 2021la; 2021b; 
Garcia & Weiss, 2020; Heim, Strauss, & Meckler, 2021; 
Sawchuk & Sparks, 2020). Findings from these studies 
suggested: 


e The pandemic negatively affected academic growth; 
students were learning less in virtual settings and 
disproportionately across content areas and student 
groups (Dickler, 2021; Curriculum Research Associates, 
2021; Heim, Strauss, & Meckler, 2021). 


e Technological barriers made it harder for many students 
to stay engaged in virtual classrooms (Garcia & Weiss, 
2020; Curriculum Research Associates, 2021; Heim, 
Strauss, & Meckler, 2021; Dickler, 2021). 


e By the end of the 2020—2021 school year, fewer students 
were on grade level in reading in all grades, particularly 
in the early elementary grades (Dorn, Hancock, 
Sarakatsannis, & Viruleg, 2020; Curriculum Research 


Associates, 2021a; 2021b; Heim, Strauss, & Meckler, 
2021). 


e Fewer students were on grade level in mathematics in all 
grades, particularly in elementary and early middle 
school grades (Mcmurdock, 2021; Curriculum Research 
Associates, 2021; Heim, Strauss, & Meckler, 2021). 


e Students impacted by poverty ended the year with six 
months of unfinished learning in mathematics 
(Curriculum Research Associates, 2021b; Garcia & 
Weiss, 2020; Heim, Strauss, & Meckler, 2021). 


e About one fifth of rising high school seniors were not on 
track to graduate (Office of Civil Rights, 2021; 
Curriculum Research Associates, 202 1a). 


e On average, students were five months behind in 
mathematics and four months behind in reading 
compared to where they should be based on historical 
data (Sawchuk & Sparks, 2020); Curriculum Research 
Associates, 2021b). 


e More students were at risk of finishing school without 
the skills, behaviors, and mindsets to succeed in college, 
the community, or the workforce (Dorn, Hancock, 
Sarakatsannis, & Viruleg, 2020). 


The 2020-2021 data from MCPS representing the first full 
year view of academic performance during the pandemic 
showed a drop in the percent of students attaining Evidence 
of Learning framework expectations during fall 2020 and 
2021 relative to pre-pandemic years (MCPS, 2021c). These 
indicators of unfinished learning were evident throughout 
every grade level and across the content areas of literacy and 
mathematics. The need for learning recovery was particularly 
pronounced in mathematics and among students receiving 
Free and Reduced Meals Services (FARMS), special 
education services, and students identified as limited English 
proficient (LEP). 


Why summer 2021 mattered? 


Ending the school year with unfinished learning presented a 
unique challenge in 2021. This challenge was not just an 
MCPS challenge, but a challenge nationally. For example, 
educators across the country expressed concern about the 
unprecedented levels of unfinished learning they could face 
among all students in fall 2021 (Goodswell, 2021). As a 
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result, summer programs were viewed as an immediate 
opportunity to address unfinished learning and prepare 
students for a more typical year in the fall. 


Funded in large part by the recently passed $1.9 trillion ARP 
Act, summer instructional programs were expected to go 
beyond addressing learning loss and remedial work to 
provide social interactions and emotional support for students 
in every age group (North, 2021; Heim, Strauss, & Meckler, 
2021; MCPS 2021a; MSDE, 2021). As a result, this study 
examined the extent to which the MCPS summer programs 
alleviated learning disruption and contributed to students’ 
recovery in mathematics, literacy, and socioemotional 
learning skills. 


Methodology 


Research Questions 


The following four research questions were used to guide this 
evaluation. 


1. What is the demographic composition of students 
who participated in the 2021 summer programs? 


2. What impact did the summer school programs have 
on students’ academic outcomes? 


3. To what extent did the 2021 summer school 
programs mitigate students’ academic learning 
disruption? 


4. What impact did the summer school program have 
on students’ social-emotional learning (SEL)? 


Evaluation Design 


This evaluation study used a pretest posttest design and 
matched comparison group design, whereby the performance 
in mathematics and literacy was measured for the same 
students before and after summer 2021 and comparisons 
made between participants and nonparticipants when 
possible. 


To assess the immediate impact of the 2021 summer school 
programs on students’ academic outcomes, the change from 
pretest to posttest in mathematics and literacy assessment 
scores were examined for summer school participants across 
subjects, grades, and student groups, through one group 
pretest posttest design (Appendix A, Table Al). The pretest 


posttest design provides for better control of the subject 
characteristics—since scores for the same students are 
analyzed at both times. 


Furthermore, due to the fact that the summer programs were 
voluntary, a matched comparison group design was used to 
assess the extent to which summer programs mitigated 
learning disruption. A matched comparison group design 
consists of 1) summer program participants (a treatment 
group) and 2) summer program nonparticipants (a 
comparison group) whose baseline characteristics were 
similar with the participants at the beginning of an 
intervention in terms of demographic variables (.e., 
race/ethnicity and special services) and spring MAP 
performance in literacy and mathematics. The average fall 
MAP scores were then compared between participants and 
nonparticipants across subjects, grades, and student groups, 
controlling for students’ demographics and spring MAP 
performance. To address question 4, online surveys were 
distributed to 22,000 middle and high school students who 
attended the summer program. 


Study Samples 


The analytical samples for this study varied across research 
questions. Descriptions for each sample are presented below. 


2021 summer program participants (treatment group). 

The sample school participants used for question 1 consisted 
of current MCPS students who attended at least one day of 
the 2021 summer school program. The analytical sample 
for question 2 comprised summer program elementary 
participants with both pretest and posttest scores in 
mathematics (N=7,861) and literacy (N=8,911) and middle 
school participants with both pretest and posttest scores in 
mathematics (N=3,347) and literacy (N=3,148). The 
analytical sample for question 3 included participants in the 
transition grades—Grade 3, 6 and, 9, who attended 75% of 
the program time and had both spring and fall 2021 MAP 
scores in mathematics (N=8,366) and/literacy (N=8,366). 
The analytical sample for question 4 is comprised of 1,466 
middle and high school summer program participants who 
responded to the online survey. 


2021 summer program Nonparticipants (the comparison 
group). The comparison group comprised a sample of 
nonparticipants who were similar to the participants in terms 
of race/ethnicity, receipt of special services (i.e., FARMS, 
identified as LEP, and special education), and the spring 
MAP scores by subject (mathematics vs. literacy) and grade 
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(i.e., Grades 3, 6, and 9). The matched nonparticipants had 
both spring and fall 2021 MAP scores in mathematics 
(N=8,366) and literacy (N=8,366), respectively (Appendix 
A, Table IC). 


Variables and Measures 


Student demographic characteristics. Student information 
(i.e., race/ethnicity, FARMS, LEP, and special education) 
was acquired from 2021-2022 enrollment records. 


Underrepresented racial/ethnic group. Students identified as 
Black or African American or Hispanic/Latino or 
traditionally underrepresented racial/ethnic backgrounds 
were combined into one group. 


Summer program participation. Office of Technology and 
Innovation (OTT) records were used to identify students who 
attended 2021 summer programs. A dichotomous variable 
was created to flag students who participated at least one day. 


Summer program attendance rate. Data on_ students’ 
absences from the summer program were used to calculate 
the attendance rate as follows: Attendance rate = (Total 
possible days - Days absent)/Total possible days. Title I 
programs lasted 24 days in total, whereas other programs 
lasted 19 days in total. 


Students recommended for summer programs. This 
measurement referred to students who were identified as 
needing summer school due to evidence of unfinished 
learning in either mathematics, literacy, or both subjects 
based on a variety of indicators. 


Academic outcomes. To examine the impact of 2021 summer 
programs, the following outcome measures were used: 


1. Summer program pretests and posttests. Grade-level 
assessments were administered to elementary and 
middle school students at the start and end of the summer 
program. 


2. MAP-Reading Fluency (MAP-RF). MAP-RF is a 
computer-adaptive reading assessment developed by 
Northwest Education Association (NWEA) for students 
in Grades Prekindergarten—5 who are learning to read. It 
measures foundational reading skills, with an emphasis 
on oral fluency (NWEA, 2021). Attainment of the Grade 
2 expectation on the spring MAP-RF was used as a 


variable to signify prior achievement in literacy for 
Grade 3 students. 


3. MAP-M and MAP-R. These tests are computer-adaptive 
mathematics and reading assessments. The Rasch Unit 
(RIT) scores obtained from MAP assessments are 
reported on a vertically equated scale. RIT scores, 
ranging from 100 to 300, were used to measure whether 
the summer programs mitigated the learning disruption 
for the program participants, compared’ with 
nonparticipants. However, there was an exception for 
Grade 3 literacy. Given that Grade 3 students took MAP- 
RF rather than MAP-R in the spring when they were in 
Grade 2, attainment of the 50" national percentile that is 
corresponding to the median fall MAP-R RIT score was 
used as an outcome measure for Grade 3 literacy in light 
of the students’ performance in the spring MAP-RF 
(attainment of grade expectation). 


4. SEL outcomes. A survey designed for middle and high 
school students was used to gather information on 1) 
skills students learned and/or strengthened through the 
summer school program, 2) perceived benefits of the 
summer school program, and 3) perceived benefits on 
social-emotional learning from the summer school 
program. 


Data Analysis Procedures 


The following analyses were conducted separately for each 
content area (mathematics and literacy), grade level, and 
student group within each school level: Kindergarten to 
Grade 2 (elementary primary), Grades 3-5 (elementary 
intermediate), middle, and high school, to answer the 
evaluation questions as applicable. 


To answer evaluation question | and 4, descriptive statistics 
were used to summarize information on the number and 
characteristics of summer program participants, program 
participation rates, and survey responses. 


To answer evaluation question 2, changes from pretests and 
posttests were examined for participants by grade, subject, 
and student group. 


To answer evaluation question 3, spring and fall MAP 
performances were examined for both participants and 
nonparticipants by grade, subject, and student group. 
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The analysis was conducted to compare summer school 
program participants with matched nonparticipants in Grades 
3, 6, and 9 over their spring and fall performance in MAP-M, 
MAP-R, and/or MAP-RF. The between-group difference in 
fall MAP performance was tested for its statistical 
significance while controlling for students’ spring MAP 
performance and demographic variables (i.e., race/ethnicity 
and recipient of special services). 


Findings 


The findings from this study are presented by evaluation 
question. Within each question, results are presented first for 
the overall population of students included in the analysis, 
then by student group if applicable. 


Question 1. What is the demographic composition of 
MCPS students who participated in the 2021 summer 
programs? 


A total of 44,552 students, representing 28.6% of students 
enrolled in the 2021-2022 school year, attended 2021 
summer programs for one or more days (Figure 1). Among 
these participants, the highest proportion was in elementary 
grades (51%, combining the primary (Grades K to 2) and 
intermediate (Grades 3-5) levels). Twenty-nine percent of 
the participants were high school students, and the remaining 
20% were middle school students (Figure 1). 


Figure 1 
Proportions of Students who Participated in 2021 Summer 
Programs by Level 
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When comparing participation rates across all grade levels, 
student participation rates ranged from as low as 24.2% for 
middle school students to as high as 34.1% for elementary 
intermediate (Grade 3 to 5) students (Figure 2). 


Figure 2 
2021 Summer Program Participation Rates by Level and 
Grade within Level 
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Participation in 2021 summer programs by student group 
and services received 


Over one-third of elementary participants received FARMS 
services (Figure 3). The majority of the elementary 
participants and over 40% of middle and high school 
participants were identified as receiving special education 
services. In addition, over one third of elementary and about 
one third of middle and high school students were identified 
as LEP students during the 2021—2022 school year. 


Figure 3 
Percentage of Participants in 2021 Summer Programs by 
Special Services 
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The participation rate by student groups varied across school 
levels, ranging from 88% among elementary Black or African 
American students receiving FARMS services to 19.9% 
among middle school Asian/White and All Other students not 
receiving FARMS services (Figure 4). 


Figure 4 
Participation Rates in 2021 Summer Programs by Students 
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Recommended students who participated in summer 
programs 


Given that the summer programs were offered for four or five 
weeks during the summer, depending on the program, 
participants and nonparticipants may have taken part in other 
programs in addition to the MCPS summer programs. Thirty- 
eight percent (N=3,020) of the 8,020 students recommended 
for summer programs attended. Overall, the lowest 
participation rates were among middle school students 
recommended for summer programs. Except for Grade 8 
(12%) and Grade 5 (28%), at every grade level, more than 
30% of the recommended students participated in the summer 
program (Figure 5). It is worthy to note that the majority of 
recommended Grade 11 students (53.6%) enrolled in the 
summer programs. 


Figure 5 
Recommended and Participated by Level and Grade within 
Level 
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Summer program attendance rate 


The overall average summer program attendance was 92%, 
indicating very high attendance at every grade and school 
level (Figure 5). The average attendance rates for the 
elementary primary and intermediate students were 86% and 
87%, respectively. Middle and high school students had 
average attendance rates of 98% and 99%, respectively. 


Question 2. What impact did the summer school program 
have on students' academic outcomes? 


Data from the pretest and posttest scores showed that Grades 
1 to 5 summer program participants experienced statistically 
significant gains in literacy assessment scores by the end of 
their summer program (P<0.05). The pretest/posttest 
improvement ranged from 6.2 percentage points for Grade 1 
to 10.3 percentage points for Grade 4 students (Figures 6 and 
7). 
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Figure 6 
Pre-Posttest Literacy Assessment Scores from Grades I—5 
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Figure 7 
Mean Percentage-Point Change in Literacy Scores from Pre 
to Posttest (Elementary) 
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Middle school students used ExactPath, an external 
assessment, scored on a scale of 500-1500, to assess the 
posttest summer program change in literacy skills 
(edmentum, 2021). Overall, middle school students made 
pre-posttest gains ranging from 0.4 to 5.1 points, in literacy 
assessments by the end of the summer program. 


Figure 8 
Mean Point Change in Middle School Literacy Scores from 
Pretest to Posttest 
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The pre to posttest gains in literacy performance fluctuated 
from 0.4 for Grade 7 to 5.1 for Grade 8 students (Figure 8). 
The growth in posttest literacy performance for middle 
school students was statistically significant for students in 
grades 6 and 8 (P<0.05). 


Change in elementary and middle school students’ literacy 
pretest to posttest scores by student groups 


The posttest literacy scores for elementary students identified 
as LEP and students receiving FARMS or special education 
services increased relative to their performance on the pretest 
(Figure 9). However, by the end of their summer program, 
there were decreases in the posttest scores at the middle 
school level relative to pretest scores for students identified 
as LEP and among students receiving FARMS or special 
education services (Figure 10). 
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Figure 9 

Mean Percentage-Point Change in Elementary Literacy 
Scores from Pre to Posttest by LEP and Students Receiving 
Special Services 
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By the end of the summer program, the literacy performance 
on the posttest for middle school students was mixed 
depending on the student group—teflecting either increased, 
comparable, or decreased performance relative to the pretest. 
The highest gains were observed for Asian/White and All 


Other Student Groups students not receiving FARMS 
services. For the remaining student groups, the posttest scores 
were comparable to their pretest scores, indicating no change. 


Scores for middle school students in the African 
American/Black and FARMS Latino/Hispanic — student 
groups decreased slightly relative to their pretest scores. This 
finding suggests that the summer program was insufficient to 
significantly change the literacy skills measured by the 
ExactPath pre and post-assessments for middle school 
students in these student groups. 


Did elementary and middle school students' mathematics 
performance change at the end of the summer school 
program? If yes, by how much? 


Yes. Students made statistically significant gains from pre to 
posttest in mathematics performance at every grade level 
assessed (p<0.05). 


Figure 11 
Pre/Posttest Mathematics Scores by Grade 
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Students' improvements based on percentage point change 
from pre to posttest mathematics assessments varied across 
grade levels, from 8.6 for Grade 8 students to 22.0 for Grade 
2 students (Figure 12). 
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Figure 12 
Mean Percentage-Point Change from Pre to Posttest 
(Mathematics) 
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Changes in mathematics pre-posttest scores by student 
group. 


The mathematics performance varied by student groups and 
services. The gains pretest to posttest for students receiving 
services were comparable across service groups, ranging 
from 16.0 to 17.7 percentage points. 


These improvements in mathematics performance were 
greater among elementary school students receiving special 
services, ranging from 17.3 to 20.8 percentage points, than 
gains made by their counterparts in the middle school. For 
middle school, the gains ranged from 9.1 to 10.8 percentage 
points. 


The highest pretest to posttest gains (20.8) were observed at 
the student group level for the Hispanic/Latino students not 
receiving FARMS services. The lowest gains were among 
middle school Asian/White and All other Groups students 
receiving FARMS services. 


Figure 13 
Mean Pre-Post Percent Point Change in Mathematics Scores 
by Student Groups 
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Question 3. To what extent did the 2021 summer school 
programs mitigate students' academic learning 
disruption? 


To investigate the extent to which the summer programs 
mitigated learning disruption, the analyses compared fall 
MAP RIT scores in mathematics and literacy for Grades 3, 6, 
and 9 summer program participants with a matched sample 
of nonparticipants. 


The expectation was that, by attending summer school, 
students would maintain and/or boost their spring 
performance, thus alleviating learning loss during the 
summer break. 


Did the summer school program participants’ fall 2021 
academic outcomes in mathematics differ from students who 
did not’ participate? If yes, by how much? 


No. The results are presented for all students as well as for 
students of the underrepresented racial/ethnic groups 
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(Black/African American and Hispanic or Latino) and 
students receiving special services within each of the grade 
levels. 


Performance on fall MAP-M assessments 


As the data in Figure 14 indicate, there was no statistically 
significant difference observed in the adjusted fall MAP-M 
RIT scores between summer program participants who 
attended 75% of the time and nonparticipants (Figure 14). 
The performance between participants and nonparticipants in 
Grade 3, 6, and 9 was similar. 


1. Grade 3: Participants and nonparticipants maintained 
the mean MAP-M RIT scores of 187 and 188, 
respectively, from spring to fall (Figure 14). 


Grade 6: Participants and nonparticipants attained a 
mean RIT score of 218 on the spring MAP-M, and these 
RIT scores decreased by three or four points in the fall 
(Figure 14). 


Grade 9: A three-point spring-to-fall decrease in mean 
MAP-M RIT scores was observed for participants (234— 
231) and a two-point decrease was observed for 
nonparticipants (232—230) (Figure 14). 


Figure 14 
Spring and Fall 2021 Mean MAP-M RIT Scores, by Grade 
Level 
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Fall 2021 Academic Outcomes in mathematics for 
Participants and Nonparticipants by Students Groups. 


Grade 3. Comparisons of the adjusted mean mathematics fall 
RIT scores between Grade 3 participant and nonparticipant 
groups showed no statistically significant differences for 
students identified as LEP and students receiving FARMS or 
special education services. (Figure 15). 


Figure 15 

Grade 3 Spring and Fall 2021 MAP-M Mean RIT Score for 
Students Identified as LEP and Students Receiving Special 
Services, by Summer Program participation Status 
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Grade 6. Across the Grade 6 students receiving special 
services, participants and nonparticipants maintained same or 
similar (one- or two-point decrease) scores from spring to fall 
(Figure 16). 


For students in the traditionally underrepresented student 
groups, summer program participants as well as_ their 
nonparticipant peers had an average score of 208 in the spring 
and dropped by two points in the fall. 
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Figure 16 

Grade 6 Spring and Fall 2021 MAP-M Mean RIT Scores for 
Students Identified as LEP and Students Receiving Special 
Services 
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Grade 9. Among _ students in_ the _ traditionally 
underrepresented racial/ethnic groups, participants and 
nonparticipants had a mean MAP-M score of 221 in the 
spring and ended with a two-point lower score of 219 in the 
fall. A similar downward trend was observed for participants 
and nonparticipants across all service groups (Figure 17). 


Figure 17 

Grade 9 Spring and Fall 2021 MAP-M Mean RIT Scores for 
Limited English Proficient and Students Receiving Special 
Services 
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Is it a concern that summer program participants did not 
outperform their nonparticipant peers on fall MAP 
mathematics performance and literacy assessments? 


No. In fact, the results suggest the summer school program 
had a positive impact in reducing loss of learning and 
academic skills over the summer break. Recent research 
examining the “summer slide” phenomenon suggested that 
on average, students lose two to three months of learning 
over the summer (Kuhfield & Tarasawa, 2020). While the 
results did not show that summer program participants 
outperformed their nonparticipant peers, the results still 
indicate the summer school program had a positive influence 
on maintaining the spring level of mathematics performance 
over the summer. 


Performance on fall 2021 MAP-R assessment 


Did summer school program participants’ fall 2021 
academic outcomes in Literacy differ from students who did 
not participate ? If yes, by how much? 


Yes. There was a significant difference in the level of 
performance of grade 3 summer school programs participants 
in literacy when compared with students who did not 
participate. 


Grade 3: In the spring, by the end of Grade 2, similar 
proportions (30.4%) of participants and nonparticipants in the 
analytical sample met or exceeded the Grade 2 expectations 
on the MAP-RF assessment. By fall, the majority of summer 
program participants (53.5%) as well as nonparticipants 
(60.2%) attained the 50" national percentile or higher on the 
fall MAP-R assessments. This difference in rates of attaining 
the 50" percentile score on fall MAP-R was significant in 
favor of the nonparticipants (p<0.05) (Figure 18). 
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Figure 18 

Percentage of Grade 3 Students who Attained Spring MAP- 
RF Grade Expectation and 50" Percentile Rank for Fall 
MAP-R Assessment 
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Grade 6: Participants and nonparticipants started with similar 
mean MAP-R RIT scores in the spring (210 vs. 211). In the 
fall, the two groups increased their performance by one or 
two points (211 vs. 213) (Figure 19). 


Grade 9: The fall and spring mean MAP-RIT scores for 
Grade 9 participants (225-225) and nonparticipants (222— 
223) were similar, indicating that students were performing 
at the same level in fall as spring. 


Figure 19 


Grade 6 and 9 Mean Spring and Fall 2021 MAP-R Mean RIT 
Score by Summer Program Attendance Status 
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Compared to students who did not attend summer school, did 
summer school program participants’ fall 2021 academic 
outcomes in literacy vary by subgroups? If yes, by how much? 


Yes. 


Grade 3. The data show an upward trend from spring- to fall 
for summer program participants and nonparticipants among 
the underrepresented student groups and students receiving 
services. The spring-to-fall increase in the proportion of 
students attaining the 50th percentile score ranged between 
9.6 (special education) and 26.0 (underrepresented student 
group) percentage points across the student groups Among 
these student groups; in every instance, the spring-to-fall 
improvement was greater for nonparticipants than for 
participants in general (Figure 20). 


Figure 20 

Proportion of Grade 3 Who Attained Spring 2™ Grade MAP- 
RF Expectations and 50" Percentile on Fall MAP-R 
Assessment: Participants vs. Nonparticipants by Focus 
Group 
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While comparing within student groups, additional analyses 
detected statistical significance in differences between the 
proportions of participants and nonparticipants who attained 
the 50" percentile on the fall 2021 MAP-R. The results 
indicated that the performance levels were similar (not 
statistically different) between participants and 
nonparticipants among students receiving FARMS or English 
language services: 
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e In the fall, 34% of participants receiving FARMS 
services attained the 50" percentile score on fall 
MAP-R, compared with 35% among 
nonparticipants. 

e In the fall, 30% of participant LEP students attained 
the 50" percentile score on fall MAP-R compared 
with 28% of nonparticipants ( Figure 20). 


However, the difference between the performance of 
participants and nonparticipants was statistically different 
among underrepresented students and students receiving 
special education services. In the fall, 39% of participants 
among the underrepresented students attained the 50° 
percentile score on fall MAP-R compared to about 44% 
among nonparticipants. In the fall, 19% of participants 
among students receiving special education services attained 
the 50" percentile score on fall MAP-R compared to about 
36% among nonparticipants. 


Grade 6. Similar to the pattern for the entire grade, the 
participants and nonparticipants identified as traditionally 
underrepresented student groups and students receiving 
services made gains of one to three RIT scores from spring to 
fall. (Figure 21) 


Grade 9. Across the student groups, participants and 
nonparticipants alike, either maintained or increased their 
MAP score by only one to two points from spring to fall 
(Figure 22). 


Figure 21 

Grade 6 Spring and Fall Mean MAP-R R RIT Scores for 
Students Identified as LEP and Students Receiving Special 
Services 
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Figure 22 
Grade 9 Fall Mean MAP-R RIT Scores for Students Identified 
as LEP and Students Receiving Special Services 
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Question 4. What impact did the summer school program 
have on students’ social-emotional learning (SEL)? 


To better understand the impact of the summer school 
program had on students SEL and developmental skills (e.g., 
critical thinking, problem solving, communication, time 
management, and communication), responses from 1,466 
students who completed the survey were analyzed. 


Positive impact in supporting students SEL 


The results from the survey data suggest the summer school 
program at MCPS had a positive impact on students’ 
socioemotional skills. Over four fifths (89%) percent of 
student survey respondents cited that the summer school 
program provided a safe and positive learning environment 
(Figure 23). In addition, when examining the students’ 
responses by grade level, a six percent difference was 
observed between high school and middle school students, 
indicating that a higher proportion of the middle school 
summer school program experienced a positive learning 
environment (Figure 23). 
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Figure 23 
Survey Results Examining Student’s Perception towards the 
Summer School Programs Learning Environment. 
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Improved students’ relationship building and collaboration 
skills 


The results from the survey data show that the summer school 
program at MCPS also had a positive impact on developing 
students’ interpersonal skills such as building positive 
relationships and improving their collaboration skills. For 
example, two thirds of the student survey respondents (66%) 
cited that they learned strategies to build positive 
relationships and work collaboratively in group settings 
(Figure 24). The data revealed that there was a significant 
difference (20 percent) between middle school students’ 
experiences when compared to their high school 
counterparts. Additional follow up research is recommended 
to better understand this phenomenon. 


Figure 24 
Percent of Students who felt They Improved Their 
Relationship and Collaboration Skills 
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Reported levels of learning and skill development by level 


To help further assess the impact summer school had on 
students learning and development, the student survey asked 
respondents to indicate whether they learned or strengthened 
the following five skills (problem solving, communication, 
collaboration, critical thinking, and time management) via 
the summer school program. The results of the survey 
findings are shown in Figure 25. 


Figure 25 
Percent of students who cited they learned or strengthened 
the following skills 
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Overall, a higher proportion of middle school survey 
respondents reported that they learned or strengthened more 
skills over the course of the summer school program than 
their high school classmates. As shown in Figure 25, more 
middle school students expressed that they felt they learned 
or strengthened their problem solving (411%), 
communication (+28%), and collaboration skills (+36%) 
when compared to high school student survey respondents. 
The responses reversed when students were asked about 
summer program impact on time management skills. Twenty 
percent more high school students indicated that they learned 
or enhanced this skill compared to middle school survey 
respondents. 


Did the summer school program have a positive impact on 
students SEL and skill development? 


Yes. The evidence from the student survey suggests that the 
summer school program had a positive influence on students 
SEL outcomes. In addition, the survey data also suggest that 
the summer school program had a positive influence in 
facilitating and enhancing student’s development in different 
skills (e.g., problem solving, communication, critical 
thinking). For example, 54 percent of all summer school 
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participants indicated that they learned or enhanced their 
ability to solve problems, and nearly 50 percent of student 
survey respondents cited that the summer school program 
helped improve or enhance their critical thinking skills 
(Figure 26). While the magnitude of impact varied by level, 
the data suggests that the summer school program was more 
beneficial and effective for middle school students. 


Figure 26 
Percent of students who cited they learned or strengthened 
their problem solving and critical thinking skills 
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Limitations 


One limitation of the study is the voluntary nature or self- 
selection of the summer program participants. Although we 
created a matched sample of nonparticipant peers, the 
matching does not capture any information on how the 
nonparticipants spent their summer. In matching, researchers 
can only ensure equal distribution of characteristics of 
students for which data are available. Therefore, caution is 
needed when interpreting the data; one cannot generalize that 
nonparticipants did not participate in any instructional 
programs similar to the MCPS summer programs. 


Another limitation is that the data related to attendance did 
not specify the number of days each student attended per se, 
but arecord of absences was available. Therefore, attendance 
rates were derived indirectly using records of absences. As 
such, analyses on how summer program daily attendance 
influenced student performance on posttest summer program 
assessments were not completed, as intended. 


The pre-posttest assessment data had inconsistencies in that 
they included data only for students with records on the Unify 
Performance Matters (PM) systems. In addition, the 
participant and nonparticipants samples are composed of 
took both the spring and fall 2021 
administration of the MAP assessments. Notably, this 


students who 


restriction excludes students who may have benefited from 
summer programs but did not have MAP scores for either fall 
or spring. 


Conclusion 


Recognizing COVID-19 impacted many students and created 
missed learning opportunities, MCPS offered voluntary 
summer programs to meet students’ educational recovery and 
socioemotional learning needs. Overall, the immediate gains 
from the summer programs as gauged through the pretest and 
posttest during the summer school program were statistically 
significant across elementary and middle school grade levels 
included in the analysis, except for Grade 7 literacy. These 
results indicated that the summer program helped increase 
students' knowledge and skills in mathematics and literacy, 
contributing to the recovery of unfinished learning. Similarly, 
the findings from the student surveys showed that students 
reported many socioemotional benefits. Over one half of the 
respondents reported that they learned or strengthened their 
problem-solving (54%) and time management skills (51%). 
Close to one-half conveyed that attending the summer 
program strengthened their critical thinking skills. 
Furthermore, over four-fifths (89%) reported that their 
summer school program provided a positive learning 
environment, and two-thirds (66%) reported they learned 
strategies to build positive relationships and collaboration. 


The expectation was that summer programs would combat 
learning loss during the break and accelerate learning toward 
recovery of any unfinished learning from the prior year. 
Overall, the majority of program participants maintained or 
improved on their spring MAP performance in mathematics 
and literacy. When compared with peers who did not attend 
the summer programs, the participants were performing at the 
same level as nonparticipants on the fall MAP mathematics 
and literacy assessments. Notably, the finding that 
participants and nonparticipants performed at the same level 
on fall MAP assessments did not necessarily indicate that the 
summer programs did not contribute to recovery and 
accelerate learning of academic knowledge and skills. Re- 
establishing strong teacher-student relationships, fostering 
time management and critical thinking skills, building 
positive relationships, and collaboration are essential to 
supporting students’ learning recovery. Researchers point out 
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that even when there was no substantial solid evidence of 
effectiveness, there is often suggestive evidence that 
voluntary programs yield benefits in reading and/or 
mathematics achievement (Bakle, 2010; Concentric Research 
and Evaluation, 2018). In any case, in the absence of data on 
how the nonparticipants spent their summer, it is important to 
note the participants may have participated in activities 
similar to those offered in MCPS schools. 


Discussion 


Taken as a whole, the findings from this study align with 
reports that few summer programs are effective in meeting all 
measured outcomes (National Academy of Sciences, 2019; 
Augustine, 2016; McCough et al., 2019; Xie, Neitzel, 
Cheung, & Slavin, 2021; Barshay, 2021). The majority of 
programs studied (about 75 percent) improve at least one 
outcome (McCombs, Augustine, Pane, Schweig, 2020; 
Hegarty, 2020). The most robust and widely cited 
randomized control study on summer learning was designed 
by researchers at RAND, a nonprofit research organization 
drawing upon expert opinions and academic theories 
(Augustine et al., 2016). The project investigated the impact 
of summer academic programs on fall performance and 
beyond in five urban school districts. With one small 
exception of the performance of Grade 4 students in 
mathematics, the results were disappointing. There was no 
evidence that the students who attended the summer 
programs did any better academically or by social-emotional 
measures than their counterparts who did not attend. 
Specifically, there was no causal evidence that the summer 
programs produced benefits in language arts, social- 
emotional outcomes, or student attendance or grades during 
the school year. 


Unless considerations about the fidelity of implementation 
are made, it cannot be determined whether a lack of explicit 
impact using rigorous post-summer academic measures is 
due to inconsistent implementation or inadequacies inherent 
in the summer programs themselves (Stains & Vickrey, 
2017). Evidence-based practice assumes that an intervention 
such as the summer programs was implemented in full 
accordance with intended goals and design. As_ such, 
researchers often explain that typically summer programs do 
not accomplish the purpose of raising literacy or mathematics 
achievement because summer programs 1) do not engage 
students sufficiently and 2) are not well run (Xie, C., Neitzel, 
A., Cheung, A., & Slavin, R. E, 2021; Barshay, 2021). 
Ensuring fidelity assumes a strong theory of action and a 
consensus on specific outcomes that a summer program can 
reasonably accomplish in a short period. 


In addition to taking into account the fidelity of 
implementation in the analysis of the effectiveness of 
summer programs, researchers point to several factors that 
are correlated with the efficacy of summer programs. For 
reading and language arts, the quality of instruction (focused 
on explicit instruction, on-task behavior, and teachers' 
ensuring that all students understood the material) was 
correlated with strong evidence of program effectiveness on 
academic achievement (Augustine et al., 2016). To provide 
sufficient content to demonstrably improve — student 
achievement, researchers recommend that districts offering 
voluntary summer programs: 1) target the needs of 
participants, 2) have programming linked to desired, clearly 
defined academic outcomes, 3) be of sufficient duration (at 
least five weeks, preferably six) of full-day programming, 
and 4) document strong attendance (Augustine et al., 2016; 
National Academies of Sciences, 2019; McCombs, 
Augustine, Pane, Schweig, 2020; Hegarty, 2020). 


In this study, the pre-posttest assessments—developed by 
district personnel and the standardized assessments used for 
the middle school literacy pre and posttest and the fall MAP 
assessments by NWEA are fundamentally different in scope, 
structure, and purpose. By design, the end-of-program 
measures are aligned and sensitive to changes in mathematics 
and literacy skills resulting from the program. The nationally 
normed standardized assessments are broader in scope and 
are not customized to a local instructional program. 
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Montgomery County Public Schools 


Office of Shared Accountability 


PROGRAM 


Local Summer School 
Program (Non-Title | 
and Non-Summer UP) 


Summer UP 


Appendices 


Appendix A Table Al 


DESCRIPTION 


School-based programs combining 
mathematics, literacy and specials/ 
enrichment 


Same as local summer school 
programs with enhanced 
programming in six elementary 
schools (Fox Chapel, Highland View, 
Oakland Terrace, Pine Crest, Rock 
View, Stedwick) and four middle 
schools (A. Mario Loiederman, 
Argyle, Gaithersburg, Neelsville). 


FORMAT 


In-person, with rotational models as 
needed based on enrollment 


4.5 hours daily; available at all 
elementary and middle schools; 
students are encouraged to attend 
at their home school; schools that 
offer ELO-SAIL will offer in-person 
and virtual options based on family 
preference 


In-person, with rotational models as 
needed based on enrollment 


Extended Learning 


Same as above with enhanced 


In-person or virtual based on family 


LENGTH 


4 Weeks: 
July 6-30; 


July 6-Aug. 6 
in Title | schools 


3-4 Weeks: 
July 6-30 
Window 


5 Weeks: 


Opportunities (ELO) programming in Title | schools preference July 6-August 6 
SAIL 
NEW! MCPS Virtual Central all-virtual programs All virtual | 4 Weeks: 
Summer School combining mathematics, literacy and July 6 - July 30 
specials/enrichment 
Extended School Year | Special education for qualified In-person or virtual based on family 4 or 5 Weeks: 
(ESY) students preference Starting July 6 
Appendices 
Applied Research and Evaluation 20 Impact of 2021 Summer Programs 


Table la 


2021 Summer Program One Group Pretest-Posttest Design 


Group: 


1st observation (pre-summer 
program measurement) of 
Mathematics, English language arts 
(ELA) (O1) 


Attended summer school (X) 
(independent variable) 


2nd observation (post-summer 
measurement) of Mathematics, 
English language arts (ELA), (O2) 


2021 Summer program 
participants 


the average score on Summer program 
pre-test assessments 


X (19 days of summer program) 


the average score on Summer program 
posttest assessments 


Table 1b 


Fall Outcomes Matched Comparison Group Design 


Group: 


1st observation (pre-summer 
program measurement) of 
Mathematics and Literacy (01) 


Attended summer school (X) 
(independent variable) 


2nd observation (post-summer 
measurement) of Mathematics and 
Literacy (O2) 


2021 Summer program 
participants 


The average score on spring 2021 MAP 
mathematics and literacy RIT scores 
achievement measures and proportion 
of students performing at 50th 
percentile on spring 2021 MAP 
assessments. 


X (19 days of summer program) 


the average score on fall 2021 MAP 
mathematics and literacy achievement 
measures and proportion of students 
performing at 50th percentile on fall 
2021 MAP mathematics and literacy 


Nonparticipants 


The average score on spring 2021 MAP 
mathematics and literacy RIT scores 
achievement measures and proportion 
of Grade 3 students performing at 50" 
percentile on spring 2021 MAP 
assessments. 


the average score on fall 2021 MAP 
mathematics and literacy achievement 
measures and proportion of Grade 3 
students performing at 50th percentile 
on fall 2021 MAP mathematics and 
literacy 
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Table 1C. Profile of Analytical Samples for Question 3 


N students included in 


Percent of Students Included in MAP-M Analysis by Grade and Student Group 


analysis % Underrepresented Students % FARMS % Special Education % LEP 
Non- 
MAP-M participants | Participants | Non-participants | Participants | Non-participants | Participants | Non-participants | Participants | Non-participants | Participants 
Grade 3 3859 3859 56.4 62.3 43.0 46.9 11.2 16.0 33.1 34.7 
Grade 6 3029 3029 53.4 57.7 40.8 42.5 17.2 21.1 26.8 28.7 
Grade 9 1478 1478 52.9 51.8 39.0 35.1 23.3 25:2 16.4 15.2 
Non- 
MAP-R participants | Participants | Non-participants | Participants | Non-participants | Participants | Non-participants | Participants | Non-participants | Participants 
Grade 3 3809 3809 53.2 62.4 41.5 47.1 78 15.7 26.4 34.9 
Grade 6 3001 3001 54.6 57.4 40.3 42.0 13.6 21.6 30.2 29.0 
Grade 9 1556 1556 53.7 51.5 38.6 33.1 14.8 23.0 18.7 14.8 
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